Virtual/augmented reality transition system and method

ABSTRACT

A system and method of operating an audio visual system generating a virtual immersive experience may include an electronic user device in communication with a tracking device that may track a user&#39;s physical movement in a real world space and translate the tracked physical movement into corresponding movement in the virtual world generated by the user device. The system may detect when a user and the user device are approaching a boundary of a tracking area and automatically initiate a transition out of the virtual world and into the real world. A smooth, or graceful, transition between the virtual world and the real world as the user encounters this boundary may avoid disorientation which may occur as a user continues to move in the real world, while motion appears to have stopped upon reaching the tracking boundary.

FIELD

This document relates, generally, to a virtual or augmented realitysystem.

BACKGROUND

In an immersive experience, such as an experience generated by a VirtualReality (VR) system or an Augmented Reality (AR) system, a relativelyclear boundary may exist between the immersive experience generated bythe VR/AR system, or the “virtual” world, and the environment outside ofthe virtual world, or the “real” world. Accuracy when translating auser's real world position into the virtual world may enhance the user'ssense of presence in the virtual world.

SUMMARY

In one aspect, a method of operating an audio visual system configuredto generate a virtual immersive experience may include activating atracking device and tracking a position of a user electronic device in areal world space, detecting a transition condition, and performing atransition process to transition out of the virtual world generated bythe user electronic device in response to the transition condition.

In another aspect, a method of operating an audio visual systemconfigured to generate a virtual immersive experience may includeactivating a tracking device having a defined tracking area and trackingmovement of a user electronic device the real world space, translatingthe real world movement of the user electronic device in the real worldspace into virtual movement in a virtual world generated by the userelectronic device, determining when a current position of the userelectronic device is within a threshold of a system boundary, andperforming a transition process when the current position of the userelectronic device is within the threshold of the system boundary.

In another aspect, an audio visual system may include a user electronicdevice generating a virtual world experience, the user electronic devicebeing movable within the real world space, a tracking device incommunication with the user electronic device, the tracking devicetracking a position of the user electronic device in the real worldspace and real world movement of the user electronic device in the realworld space, and a processor. The processor may translate the detectedreal world movement of the user electronic device into virtual movementin the virtual world generated by the user electronic device, mayautomatically perform a transition out of the virtual world generated bythe user electronic device when a distance between the tracking devicein the tracking area and a boundary of the tracking area is less than orequal to a preset threshold distance such that the transition out of thevirtual world is complete at or before the user electronic devicereaches the boundary, and may perform a transition back into the virtualworld by the user electronic device when the tracking device detectsthat the user electronic device has crossed the boundary and re-enteredthe tracking area.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are perspective views and FIG. 1C is a side view of ahead mounted display device, in accordance with an embodiment broadlydescribed herein.

FIG. 2 is a block diagram of the head mounted display device, inaccordance with an embodiment broadly described herein.

FIGS. 3A-3C illustrate an example implementation of a system forautomatically transitioning into and out of an immersive experience, inaccordance with an embodiment broadly described herein.

FIG. 4 illustrates another example implementation of a system forautomatically transitioning into and out of an immersive experience, inaccordance with an embodiment as broadly described herein.

FIGS. 5A and 5B illustrate another example implementation of a systemfor automatically transitioning into and out of an immersive experience,in accordance with an embodiment as broadly described herein.

FIGS. 6A and 6B illustrates example implementations of externalindicators generated by the example head mounted display shown in FIGS.1A-1C, in accordance with embodiments as broadly described herein.

FIG. 7 illustrates a pass through mode of the example head mounteddisplay shown in FIGS. 1A-1C, in accordance with an embodiment asbroadly described herein.

FIG. 8 illustrates a transparent mode of the example head mounteddisplay shown in FIGS. 1A-1C, in accordance with an embodiment asbroadly described herein.

FIG. 9 illustrates a rotation of the example head mounted display shownin FIGS. 1A-1C, in accordance with an embodiment as broadly describedherein.

FIG. 10 is a flowchart of an example process of initiating a transitionprocess, in accordance with embodiments as broadly described herein.

FIG. 11 is a flowchart of a transition process shown in FIG. 10, inaccordance with embodiments as broadly described herein.

DETAILED DESCRIPTION

A Virtual Reality (VR) system and/or an Augmented Reality (AR) systemmay include, for example, a head mounted display device (HMD) or similardevice worn by a user, for example, on a head of the user, to generatean immersive VR environment to be experienced by the user. Movement ofthe user in the real world may be translated into corresponding movementin the environment generated in the virtual world. Differences in thephysical boundaries of the real world, such as, for example, theconfines of a room and/or objects in the room, and boundaries of thevirtual world may cause discrepancies or disruptions in the immersiveexperience and disorientation as the user approaches and/or encountersone of these virtual boundaries and/or one of these physical boundaries.A smooth, or graceful, transition, for example, from the virtual worldto the real world as the user encounters a boundary of the virtualworld, may avoid the disorientation and motion sickness which may occuras a user continues to move in the real world, while motion appears tohave stopped upon reaching the boundary of the virtual world.

FIGS. 1A and 1B are perspective views and FIG. 1C is a side view of anexample HMD which may be worn by a user to generate an immersive virtualexperience. The example HMD 100 may include a housing 110 in whichoptical components may be received. The housing 110 may be coupled, forexample, rotatably coupled and/or removably attachable, to a frame 120which allows the housing 110 to be mounted or worn on a user's head. Anaudio output device 130 may also coupled to the frame 120, and mayinclude, for example, speakers mounted in headphones and coupled on theframe 120.

As shown in FIG. 1B, a front face 110 a of the housing 110 rotated awayfrom or otherwise removably attached to a base portion 110 b of thehousing 110. A display 140, such as, for example, a smartphone or otherdisplay device, may be mounted on the front face 110 a of the housing110. Lenses 150 may be mounted in the housing 110, between the user'seyes and the display 140 when the front face 110 a is in the closedposition against the base portion 110 b of the housing 110. A positionof the lenses 150 may be adjusted by an adjustment device 158, so thatthe lenses 150 may be aligned with respective optical axes of the user'seyes to provide a relatively wide field of view and relatively shortfocal length.

The HMD 100 may also include a sensing system 160 including varioussensing system devices 162-164 and a control system 170 includingvarious control system devices 171-176 to facilitate manual user controland automated control of the HMD 100. The control system 170 may alsoinclude a processor 190 to control operation of the components of thecontrol system 170, both when operated manually and independently, andalso when operated in response to conditions detected/sensed by thesensing system 160 in response to a command received by the controlsystem 170.

In some embodiments, the HMD 100 may also include a camera 180 which maycapture still and moving images of the real world environment anddisplay these images to the user on the display 140 in a pass throughmode. In the pass through mode, the user may be able to leave thevirtual world and temporarily return to the real world without removingthe HMD 100 or otherwise changing the configuration of the HMD 100 suchas, for example, moving the housing 110 out of the line of sight of theuser. In some embodiments, the display 140 may be a transparent display,allowing the user to view the real world environment through the display140 in some modes of operation.

A block diagram of a system 200 for transitioning from avirtual/augmented reality environment to an ambient environment in shownin FIG. 2. The system 200 may include a user electronic device 250, suchas, for example an HMD as described above with respect to FIGS. 1A-1C,to generate the virtual reality environment, in communication with atracking system 240 that tracks a user position in a real world,physical space, so that the user's physical movement in the real worldspace may be translated into movement in the virtual reality world bythe system 200. The device 250 may include a sensing system 260 and acontrol system 270, which may be similar to the sensing system 160 andthe control system 170, respectively, shown in FIGS. 1A-1C. In theexample embodiment shown in FIG. 2, the sensing system 260 includes alight sensor 262, a distance/proximity sensor 263 and an audio sensor264, and the control system 270 includes a power control device 271, apause control device 272, a manual audio control device 273, a manualvideo control device 274, a rotation control device 275 to controlrotation of the housing relative to the frame, an optical adjustmentcontrol device 258 to adjust optical components of the device 250, and atransition control device 276 to control transitions between the virtualenvironment and the ambient environment. In some embodiments, thesensing system 260 and/or the control system 270 may include more, orfewer, devices, depending on a particular implementation. The elementsincluded in the sensing system 260 and/or the control system 270 canhave a different physical arrangement (e.g., different physicallocation) within, for example, an HMD other than the HMD 100 shown inFIGS. 1A-1C.

In a VR system, a user may physically move in a prescribed physicalspace in which the system is received and operated. The system may trackthe user's movement in the physical space, or the “real” world, andcause the virtual world to “move” in coordination with the user'smovement in the real world. This positional tracking may thus track aposition of a user in the real world and translate that movement intothe virtual world to generate a heightened sense of presence in thevirtual world.

In some embodiments, this type of motion tracking in the space may beaccomplished by, for example, a tracking device such as a camerapositioned in the space, and in communication with a base stationgenerating the virtual world in which the user is immersed. This basestation may be, for example, a standalone computing device, or acomputing device included in the HMD worn by the user. In the exampleimplementation shown in FIG. 3A, a tracking device 310, such as, forexample, a camera 310, is positioned in a physical, real world space300, and is oriented to capture as large a portion of the space 300 aspossible with its field of view. This tracking area 320 defined by, forexample, the field of view and range of the tracking device 310, isrepresented by the shaded portion of the space 300 shown in the exampleillustrated in FIG. 3A. In this example, the tracking area 320 of thetracking device 310 is smaller than the real world space 300 in whichthe system is received. As a user A, immersed in the virtual world,moves through the real world space 300, a position of the user A in thespace 300 may be tracked by the tracking device 310 and translated intoa position/movement in the virtual world, provided the user A remainswithin the field of view, or tracking area 320, of the tracking device310. Because in this example the tracking area 320 is smaller than thereal world space 300, it is possible for the user A to move out of thetracking area 320 and still be in the space 300. If the user A moves outof the tracking area 320, or for some other reason cannot be “seen” bythe tracking device 310 (such as, for example, being blocked by anobstacle), then continued movement of the user A in the real world space3200 will not be translated into corresponding movement in the virtualworld. Rather, after tracking is lost, while the user A continues tomove in the real world space 300, the virtual world will remain still,or appear to be stuck.

As shown in FIG. 3A, the user A, immersed in the virtual world, may movein the real world space 300, for example, sequentially from position 1to position 5. At positions 1, 2 and 3, the user A is completely withinthe field of view, or tracking area 320, of the tracking device 310. Atposition 3, the user A is within the tracking area 320, but close to aboundary of the real world space 300, such as, for example, a wall. Atposition 4, the user A is just within the tracking area 320, very closeto a boundary of the tracking area 320. At position 5, the user hascrossed the boundary of the tracking area 320 and has moved completelyout of the field of view, or tracking area 320 of the tracking device310.

At positions 1, 2 and 3 shown in FIG. 3A, because the user A is withinthe tracking area 320, movement of the user A in the real world space300 tracked by the tracking device 310 may be translated intocorresponding movement in the virtual world. At position 4, the trackingdevice 310 may still track movement of the user A in the real worldspace 300 and translate that tracked real world movement into movementin the virtual world. However, due to the user's proximity to the edgeof the tracking area 320, or the point at which the tracking device 310can no longer track the movement of the user A, it may be consideredthat, at position 4 the user A is about to exceed a predeterminedthreshold at which loss of tracking may be near, or imminent, as atposition 5. That is, as discussed above, if the user A continues to movein the room 300 after tracking of the user A by the tracking device 310is lost, the virtual world may remain still, or appear stuck, while theuser A continues to move, which may cause disorientation, motionsickness, dissatisfaction with the virtual immersive experience and thelike. A fluid, or graceful, transition from the virtual world to thereal world, before actually reaching this still or stuck point in thevirtual world, may alleviate the negative effects of disorientation,motion sickness and the like.

In the example shown in FIG. 3B, a theoretical zone Z surrounding theuser A is denoted by dotted lines surrounding the user A at each of thepositions 1 through 5. Although the example zone Z shown in FIG. 3B isessentially circular, surrounding the user A, simply for ease ofdiscussion and illustration, the zone Z may have other shapes and/orextend across a different area relative to the user A. For example, thezone Z may be defined by an area extending mostly in front of the user Aas the user moves in a particular direction. Regardless of the shape orcontour of this zone Z, the zone Z may represent, for example, a warningthreshold or boundary associated with the position of the user A in thespace 300 relative to the tracking area 320 of the tracking device 310.At positions 1, 2 and 3, the user A and the zone Z surrounding the userA are completely within the tracking area 320, and physical movement ofthe user A in the space 300 may be translated to corresponding movementin the virtual world as described above. At position 4, the user A iswithin the tracking area 320, but in close proximity to the edge orboundary of the tracking area 320, with a portion of the zone Zsurrounding the user A outside of the tracking area 320. At position 5,the user A and the zone Z surrounding the user A are completely outsideof the tracking area 320.

As noted above, while at position 4, the tracking device 310 may stilltrack movement of the user A in the space 300 and translate the trackedphysical movement into movement in the virtual world. However, theuser's proximity to the edge of the tracking area 320, and the movementof a portion of the zone Z close to, or beyond, the boundary of thetracking area 320, may trigger the initiation of a fluid, or graceful,transition process from the virtual world to the real world beforetracking is fully lost and real world physical movement can no longer betranslated into movement in the virtual world. This may avoid thediscomfort and disorientation that may be associated with a more abruptend to the virtual world as the user A continues to move in the realworld.

In some embodiments, the tracking device 310 may not just track thephysical position and movement of the user A in the space 300, but mayalso detect when the user A is within a predetermined distance, orthreshold, of a boundary or edge of the tracking area 320, at which lossof tracking may be imminent. As shown in the example illustrated in FIG.3B, this loss of tracking by the tracking device 310 may be consideredimminent when, for example, the zone Z approaches the boundary of thetracking area 320, in advance of when the user actually moves into anarea of the space 300 which cannot be tracked by the tracking device310. For example, the system may issue a warning or indicator, and/ormay automatically initiate a fluid or graceful transition out of thevirtual world when the user is at position 4, even though the user isstill in the tracking area 320 at position 4, so that by the time theuser reaches position 5, the transition out of the virtual world will becomplete, and the user may avoid disorientation and/or discomfort due tothe loss of tracking.

In some embodiments, the zone Z at which the transition process isinitiated may be set based on one or more different factors such as, forexample, size/dimensions of the space 300, a type of virtual immersiveexperience generated by the HMD 100, an amount and/or speed of movementof the user in the space 300, a position of the tracking device 310,user preferences, and other such factors. In the example implementationshown in FIG. 3B, if dimensions of the space 300 were, for example, 12feet by 12 feet, with the tracking area 320 denoted by the shadedportion of the space 300 in FIG. 3B, and the zone Z at which thetransition process is initiated were set at, for example, a 3 footradius surrounding the user, based on various different factors asdescribed above, then the gradual transition out of the virtual worldmay be carried out as shown in FIG. 3C.

As shown in FIG. 3C, the user is completely within the tracking area 320at position 3A. As the user moves from position 3A to position 3B, theuser approaches the point at position 3B at which the edge of theestablished 3 foot radial zone meets the boundary of the tracking area320. As the HMD 100 and/or the tracking device 310, or other componentof the system, determines that the user is positioned within the presetthreshold (in this particular example, 3 feet of the boundary of thetracking area 320, as at position 3B), the system 100 may initiate agradual transition out of the virtual world. At position 4, the user hasmoved closer to the boundary of the tracking area 320 (for example,within 1 foot of the boundary as shown in the example illustrated inFIG. 3C). At position 4, the transition out of the virtual world may beat least partially complete, with the transition process fully completeby the time the user reaches position 5 and is fully outside of thetracking area 320.

The transition process, to transition the user from a fully immersedvirtual reality state at position 3A to a real world state at position5, may follow a variety of different profiles. Transition profiles maybe consistent with, for example, the type of immersive virtualexperience generated by the HMD 100, the speed and/or direction the useris moving in the space 300, preset user preferences, and other suchfactors. In the example implementation shown in FIG. 3C, simply for easeof discussion and illustration, if a linear transition profile isassumed, the transition process may be initiated at position 3B, whenthe user is 3 feet from the boundary (the preset zone in this example),and may be ⅔ complete when the user is positioned 1 foot from theboundary at position 4. In some embodiments, an indicator may signalinitiation of the transition process. The indicator may include, forexample, a visual indicator, an audible indicator, a physical indicatorsuch as, for example, buzz or vibration generated by the HMD 100 and thelike.

This example of a linear progression of the transition from the virtualworld to the real world is just one example of how the transitionbetween the virtual world and the real world may be accomplished. Insome embodiments, the transition between the virtual world and the realworld may follow, for example, a non-linear profile, a stepped profile,or other profile based on various different factors as noted above. Insome embodiments, the system may automatically adjust the transitionprofile, such as, for example, selecting a different transition profileand/or adjusting an acceleration rate of the profile, based on, forexample, changes in speed at which the user is approaching the boundary,changes in direction of the user, and other such factors.

In some embodiments, the tracking device 310 may detect the user'sproximity to a boundary of the tracking area 320, or that the user iswithin a threshold distance of a boundary of the tracking area 320. Insome embodiments, the HMD 100 may include a sensor that detects that theuser user's proximity to a boundary of the tracking area 320, or thatthe user is within a threshold distance of a boundary of the trackingarea 320. For example, one of the sensors of the sensing system 160(such as, for example a proximity sensor 163) and/or the camera 180 ofthe HMD 100 may detect this proximity to the boundary, and the controlsystem 170/processor 190 may initiate the transition process asdescribed above.

In the example implementation shown in FIGS. 3A-3C, the tracking area320 is smaller than the space 300 in which the system is operated,resulting in areas in the space 300 which are outside of the field ofview of the tracking device 310, and thus user movement in those areascannot be tracked. In some embodiments, the tracking area 320 of thetracking device 310 may be the same as, or greater than, the size of thespace 300 itself, based on, for example, dimensions of the space 300,capabilities of the tracking device 310, installation position of thetracking device 310 in the space 300, and the like.

In the example implementation shown in FIG. 4, the tracking device 310can capture substantially the entirety of the real world space 300 inits field of view, or tracking zone 320. In this example, tracking ofthe user is not lost due to the user leaving the tracking zone 320, and,as the user moves from sequentially from position 1 to position 5, thetracking device 310 may track the physical movement of the user in thespace 300 and that physical movement may be translated intocorresponding movement in the virtual world. To avoid obstacles in theroom which may, for example, pose a hazard to the user, a gradualtransition out of the virtual world may be performed, as describedabove, as the user approaches an obstacle. In the example shown in FIG.4, the user is within the predetermined threshold, or zone, approachinga wall of the space 300 at position 4. Thus, at position 4, the systemmay initiate a gradual transition out of the virtual world experience,so that, by the time the user reaches position 5, the transition out ofthe virtual world is complete and the user may avoid collision with thewall. A similar approach may be applied to other detecting and avoidingother obstacles, such as, for example, tables, chairs, stairs and thelike, present in the space 300. In some embodiments, the configurationof the space 300, such as, for example, the dimensions and any furnitureand the like in the room may be previously known and stored by thesystem. In some embodiments, the system may scan and store aconfiguration of the space 300 prior to initiation of a VR experience.

For example, in some embodiments, the tracking device 310 may beconfigured to scan and collect images of the space 300 to essentiallypaint, or construct, an image of the space 300. This scanning may bedone once, for example, upon initiation of a session, or may beperiodically or substantially continuously updated. In the exampleimplementation shown in FIG. 5A, the tracking device 310 tracks the userin the space 300 as the user moves from position 1 to position 2. In animplementation in which the tracking device 310 is configured tointermittently scan and re-paint the space 300, in addition to trackingthe user, the tracking device 310 may detect, for example, other objectsin the space 300 which may pose an obstacle to the user, or may detectchanges in the space 300 such as, for example, a door 315 moving fromthe closed position shown in FIG. 5A to the open position shown in FIG.5B. As the open door 315 shown in FIG. 5B may pose an obstacle, or ahazard, to the user, the system may generate a warning indicator, and/ormay initiate the transition process upon detection of the obstacle, sothat the obstacle is visible to the user and/or the user is aware of theobstacle.

As noted above, in some embodiments the system may generate a warningindicator and/or initiate the transition process upon detection ofobstacles, i.e., stationary obstacles, already present in the room, suchas, for example, furniture and the like. The warning indicator mayinclude, for example, a visual indicator and/or an audible indicatorand/or a physical indicator such as, for example a buzz or vibration ofthe HMD 100. The system may also generate a warning indicator and/orinitiate the transition process upon detection of new obstacles detectedas the tracking device 310 scans and updates the configuration of thespace 300, such as, for example the opening of the door 315 shown inFIG. 5B. In some embodiments, the system may also generate a warningindicator and/or initiate the transition process in a situation inwhich, for example, another person approaches the user. This may beimplemented to avoid collision between multiple players in a scenario inwhich multiple players or users are in the same space 300, engaged in avirtual reality immersive experience, in a situation in which anotherperson requires the attention of the user, and other such circumstances.As noted above, the rate at which the warning indicator and/or thetransition process is initiated and carried out, may be based onnumerous different factors, including, for example, the nature of thevirtual reality immersive experience, the speed and/or direction of theuser(s) in the space 300, and the like.

In some embodiments, the transition between the virtual world and thereal world may be initiated, for example, as the user approaches theboundary of the tracking zone 320, or when an obstacle is detected inthe path of the user, when another user engaged in the VR experience isin the path of the user, when another person requires the attention ofthe user, and other such situations as discussed above in more detail.In some embodiments, when transitioning from the virtual world to thereal world, the transition may include a gradual fade of the imagedisplayed on the display 140 of the HMD 100. The gradual fade mayinclude a gradual fade from the image associated with the virtual worldto, for example, a monochromatic display, such as, for example, an allwhite screen, or an all black screen and the like. In some embodiments,the transition may include the display of pass through images of thereal world environment on the display 140, captured by, for example, thecamera 180 on the HMD 100. In some embodiments, these pass throughimages may be, for example, superimposed on, or ghosted on, the virtualreality images displayed on the display 140. In some embodiments, thedisplay 140 of the HMD 100 may be a transparent display, allowing thereal world environment to be viewed through the display 140 after thevirtual images are no longer displayed.

In some embodiments, the fade out of the virtual images displayed on thedisplay 140 of the HMD 100 and the fade in of, for example, themonochromatic display, or the images of the real world captured by thecamera 180 and passed to the display 140, or the visibility of the realworld afforded by a transparent display, may be carried out in asymmetric fashion, so that as the images of the virtual world fades, thevirtual images are replaced by images of the real world at acorresponding rate. In some embodiments, the fade out of the virtualimages and the fade in of, for example, the monochromatic display, orthe images of the real world captured by the camera 180 and passed tothe display 140, or the visibility of the real world afforded by atransparent display, may be carried out in an asymmetric fashion.

In some embodiments, the rate at which the transition is performed maybe dynamically adjusted based on, for example, the type virtualimmersive experience generated by the HMD 100, the speed and/ordirection of the user, the speed, direction and/or proximity of otheruser(s), environmental differences between the virtual world and thereal world (such as, for example, differences in brightness levels,differences in sound levels), user preferences and the like.

In some embodiments, the transition may include a pause in movement oran action sequence in the virtual world, or a change in color (forexample, from full color to black and white), prior to proceeding intothe fade, to serve as an indicator that a transition is to occur.

In some embodiments, the transition may include a fade of sound content.For example, when transitioning from the virtual world to the realworld, audio content associated with the virtual world may fade out,accompanied by a fade in of ambient sound. In some embodiments, the fadeout of virtual audio content and the fade in of ambient sound mayaccompany the fade out of virtual images and the fade in of real worldimages. In some embodiments, the fade out of virtual audio content andthe fade in of ambient sound may be accomplished serially, so that theambient sound is not superimposed on, or output at the same time as, thevirtual audio content.

In some embodiments, the system may be configured to simulate otherenvironmental factors associated with the virtual world, such as, forexample, temperature, smell and the like. When so configured, the systemmay also gradually transition these other environmental factors from thevirtual world to those of the ambient environment in a similar manner.

In some embodiments, the system may be configured to detect when theuser re-enters the tracking zone 320, or has moved away from anobstacle, or two users in the same space 300 are no longer on a path tocollision, and may perform a transition from the real world back intothe virtual world in a manner similar to the process described abovewith respect to transitioning from the virtual world into the realworld. That is, in some embodiments, the transition from the real worldback into the virtual world may include a fade out of the real worldimages displayed on the display and a fade in of the virtual worldimages, in a symmetric or an asymmetric manner, In some embodiments, thetransition from the real world back into the virtual world may include aghosting of the virtual world images onto the real world images as thereal world images fade out and are replaced by the virtual world images.In some embodiments, the transition from the real world back into thevirtual world may include an indicator prior to initiating thetransition, such as, for example, a visual indicator and/or an audibleindicator and/or a physical indicator. In some embodiments, thistransition back into the virtual world may allow the user to resumeactivity in the virtual world at a point at which the transition fromthe virtual world to the real world was initiated, to avoid loss ofcontent and functionality.

As noted above, in some embodiments, the transition process may beinitiated in response to another person, or external party, approachingthe user, or otherwise attempting to gain the attention of the user whois immersed in the virtual experience generated by the HMD 100. Forexample, the HMD 100 may recognize a gesture implemented by the externalparty as a request to gain the attention of the user. This type ofgesture may be, for example, a hand wave or other such gesture capturedby camera 180 and recognized by the HMD 100 as a request to gain theattention of the user. In some embodiments, the HMD 100 may recognizethe approach of the external party, such as, for example, another personnot engaged in the virtual immersive experience, as a request to gainthe attention of the user. In some embodiments, the HMD 100 mayrecognize particular key words or sounds as a request to gain theattention of the user. The HMD 100 may initiate the transition processin response to a recognizing a request to gain the attention of theuser.

In some embodiments, when the transition process is initiated inresponse to a request from another person, or external party, to gainthe attention of the user, the HMD 100 may generate an externalindicator to confirm to the external party that transition is inprocess, or has been completed, and that the user is disengaged from thevirtual world and available for interaction with the external party, inthe real world. In some embodiments, this external indicator to theexternal party may be generated by the HMD 100, and may be, for example,a visual indicator, an audible indicator, a physical indicator such as amovement of the HMD 100, and various combinations of these types ofindicators.

In some embodiments, this external indicator may include, for example, achange in a physical appearance in some portion of the HMD 100 that maybe visible by the external party. This physical change in appearance mayinclude, for example, illumination of an indicator light on the HMD 100,such as, for example, illumination of one or more of the control systemdevices 171-176 of the control system 170, or illumination of one ormore indicator lights 181 (see FIG. 1A) provided on the HMD 100. In someembodiments, this physical change may include, for example, a change inappearance of certain portions of the HMD 100. A change in appearance ofthe HMD 100 may include, for example, a change in color of certainportions of the housing 110 and/or frame 120 of the HMD 100 achieved by,for example, one or more light sources provided in and/or on the HMD100, such as illumination of a sequence of light emitting diodes whichcan change illumination color, flash and the like.

In some embodiments, portions of the housing 110 and/or frame 120 of theHMD 100 may be made of a material that can transition between an opaquestate and a transparent state, such as, for example, a polymer dispersedliquid crystal (PDLC) material. When powered off, this type of materialmay be opaque. When activated by light, liquid crystal dropletsdispersed in the material may transmit through the material, causing thematerial to transition from an opaque state to a clear, or transparentstate as shown in FIG. 6A. A portion of the housing 110 and/or the frame120 of the HMD 100 made of this type of material may affect a change inthe physical appearance of the HMD 100 which may be visible to theexternal party as an indicator that the user is disengaged from thevirtual world and available for interaction in the real world.

In some embodiments, a change in physical appearance of the HMD 100visible to the external party may include, for example, a change inappearance of an externally visible side of the display 140, as shown inFIG. 6B. For example, in some embodiments, the display 140 may be asmartphone removably coupled in the housing 110, or other electronicdevice including a display surface visible to the user from an internalfacing side of the device, and a display surface visible to the externalparty from an external facing side of the device. In this type ofarrangement, an image may be displayed on the external facing side ofthe device, which may be visible to the external party as an indicatorthat the user is disengaged from the virtual world and available forinteraction in the real world. The image displayed to the external partymay include, for example, a simple change in color of the externalsurface of the device, a message in the form of characters and/or icons,an image of a set of eyes to simulate the user returning a gaze to theexternal party, and the like.

In some embodiments, the transition in response to the request from theexternal party to gain the attention of the user may also include thedisplay of pass through images to the user. These pass through imagesmay be captured by the camera 180 and displayed on the display 140 ofthe HMD 100, as shown in FIG. 7.

In some embodiments, the display 140 may be a transparent display. Inthis case, in response to a request from an external party to gain theattention of the user, the control system 170 may initiate thetransition out of the virtual world, and cause the display 140 totransition to the transparent state as the virtual world images fade orare otherwise no longer displayed, as shown in FIG. 8. In thetransparent state, the user may view the external party directly throughthe display 140, and the user's eyes may be visible to the externalparty through the transparent display as shown, for example, in FIG. 6A,indicating that the user is disengaged from the virtual world andavailable for interaction in the real world.

In some embodiments, in response to a request from an external party togain the attention of the user, the control system 170 may activate atransition control module 176 to initiate the transition out of thevirtual world. Activation of the transition control module 176 may causethe housing 110 (and the optical components housed therein) to rotatewith respect to the frame 120, as shown in FIG. 9, from the down/closedposition to the open position, so that the housing 110 and opticalcomponents are out of the line of sight of the user, so that the userhas an unobstructed view of the real world, and the user's eyes arevisible to the external party.

As noted above, in some embodiments, the transition from the virtualworld to the real world, or from the real world into the virtual world,may follow a particular profile based on, for example, a type of virtualimmersive experience, a direction and/or speed of the user, userpreferences, and other such factors. The transition may also take intoaccount environmental factors such as, for example, a brightness leveland a sound level in the real world space 300 compared to a brightnesslevel and a sound level in the virtual immersive experience generated bythe HMD 100. For example, in some embodiments, a brightness level in thespace 300 may be sensed by a light sensor 162 of the sensing system 160,or by a light sensor included in the camera 180, and a sound level inthe space 300 may be sensed by an audio sensor 164 of the sensing system160, or by a microphone included in the camera 180. When transition isto be initiated, the brightness level and the sound level of the realworld space 300 sensed by the sensing system may be compared to thebrightness level and sound level of the virtual world by the processor190, and the transition process, for example, the transition profile,may be selected/established taking into account the degree of differencebetween these environmental factors in the virtual world and the realworld.

FIG. 10 is a flowchart of a process for initiating a transition betweena virtual environment and an ambient environment, in accordance withembodiments as broadly described herein. As shown in FIG. 10, at block1010 the user may activate the system, including the HMD 100, to allowthe HMD 100 to generate the virtual immersive experience and thetracking device 310 to track the user in the space 300 as described indetail above. If, at some point during operation of the system, atransition command is received, as at block 1020, the transition processmay be initiated at block 1050. As described in detail above, thetransition command may be, for example, a direct request input by theuser, for example via the control system 170 of the HMD 100 or otheruser gesture or keyword recognized by the HMD 100. In some embodiments,the transition command may be a gesture, or keyword, or other such inputfrom an external party that is recognized by the HMD 100. In someembodiments, the transition command may be, for example, detection of anobstacle in the path of the user that triggers transition out of thevirtual world, as described in detail above.

In the absence of a specific command to transition from the virtualworld, the HMD 100 and/or the tracking device 310 may determine acurrent position of the user in the tracking area 320 at block 1030, asdescribed above. If it is determined, at block 1040, that the user iswithin a set threshold or range of the boundary of the tracking area320, at which loss of tracking by the tracking device 310 may beimminent, the transition process may be initiated at block 1050. Thetransition process is shown in more detail in FIG. 11.

As shown in FIG. 11, a current distance between the user and theboundary of the tracking area 320 may be detected, and may be comparedto a previously detected distance between the user and the boundary, atblock 1051. Based on this comparison, a direction and/or speed and/oracceleration of the movement of the user may be determined, at block1052, to confirm the user is approaching the boundary at block 1053. Ifit is confirmed at block 1053 that the user is approaching/nearing theboundary (and is about to enter an area where the tracking device 310may not be able to continue to track the user's physical movement in thespace 300), a transition profile may be implemented, at block 1054,based on various factors, such as, for example, the virtual environment,the real world space 300 in which the virtual environment isimplemented, direction and/or speed and/or acceleration associated withthe movement of the user, and other such factors.

If, at block 1055, it is determined that the transition profile includesvideo parameters, then, at block 1056, the virtual image(s) associatedwith the virtual world displayed on the display 140 of the HMD 100 maybe replaced with transition image(s) displayed on the display 140 inaccordance with the profile. As discussed above, this replacement of thevirtual image(s) with transition image(s) may include, for example, asymmetric or asymmetric fade out of the virtual image(s) accompanied bya fade in of the transition image(s). The transition images may include,for example, a monochromatic display, a display of text and/orcharacters and/or icons, a ghost image of the real world superimposed onthe virtual image, a pass through image, and the like, as discussed indetail above. This transition may also include a transparent display 140transitioning to a transparent state so that the real world is visibleto the user through the transparent display, or a movement of a portionof the HMD 100 out of the field of view of the user so that the realworld is directly visible to the user.

If, at block 1057, it is determined that the transition profile includesaudio parameters, then, at block 1058, the virtual audio outputassociated with the virtual world and output to the user via the audiooutput device 130 of the HMD 100 may be replaced with transition audiooutput in accordance with the profile. As discussed above, thisreplacement of the virtual audio output with transition audio output mayinclude, for example, a serial fade out of the virtual audio followed bya fade in of the transition audio output. The transition audio outputmay include, for example, the actual sound generated in the real worldspace 300, a preset tone, no sound, and the like.

As noted above, environmental factors in the virtual world and in thereal world may be taken into account in the transition process, so as toease the transition between the virtual world and the real world, toenhance user comfort during transition. Once each of the differentfactors/parameters included in the implemented transition profile havebeen transitioned, the transition process may be completed at block1059.

As noted above, after the transition process has been completed, the HMD100 and/or the tracking device 310 may also detect when the userre-enters the tracking area 320 and may transition from the real worldback into the virtual world. In some embodiments, this transition fromthe real world back into the virtual world may allow the user tore-enter the virtual world at a point at which the transition out of thevirtual world was initiated, allowing the user to essentially pick upwhere he/she left off.

In a system and method, in accordance with embodiments as broadlydescribed herein, a transition between the virtual world and the realworld my be automatically carried out based on a user's position in atracking area of a space in which a tracking device may track the user'sphysical movement in the space and translate that physical movement intocorresponding movement in the virtual world. Automatically initiatingthis transition before tracking of the user's physical movement is lostmay avoid the disorientation and discomfort associated with loss oftracking and corresponding loss of movement in the virtual world.

Implementations of the various techniques described herein may beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. Implementations mayimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine-readablestorage device (computer-readable medium), for processing by, or tocontrol the operation of, data processing apparatus, e.g., aprogrammable processor, a computer, or multiple computers. Thus, acomputer-readable storage medium can be configured to store instructionsthat when executed cause a processor (e.g., a processor at a hostdevice, a processor at a client device) to perform a process.

A computer program, such as the computer program(s) described above, canbe written in any form of programming language, including compiled orinterpreted languages, and can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program can bedeployed to be processed on one computer or on multiple computers at onesite or distributed across multiple sites and interconnected by acommunication network.

Method steps may be performed by one or more programmable processorsexecuting a computer program to perform functions by operating on inputdata and generating output. Method steps also may be performed by, andan apparatus may be implemented as, special purpose logic circuitry,e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the processing of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor for executing instructions and one or more memorydevices for storing instructions and data. Generally, a computer alsomay include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic, magneto-optical disks, or optical disks. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in special purposelogic circuitry.

To provide for interaction with a user, implementations may beimplemented on a computer having a display device, e.g., a cathode raytube (CRT), a light emitting diode (LED), or liquid crystal display(LCD) monitor, for displaying information to the user and a keyboard anda pointing device, e.g., a mouse or a trackball, by which the user canprovide input to the computer. Other kinds of devices can be used toprovide for interaction with a user as well; for example, feedbackprovided to the user can be any form of sensory feedback, e.g., visualfeedback, auditory feedback, or tactile feedback; and input from theuser can be received in any form, including acoustic, speech, or tactileinput.

Implementations may be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation, or any combination of such back-end, middleware, orfront-end components. Components may be interconnected by any form ormedium of digital data communication, e.g., a communication network.Examples of communication networks include a local area network (LAN)and a wide area network (WAN), e.g., the Internet.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, the appearances of the phrase “in oneembodiment” or “in an embodiment” in various places throughout thisspecification are not necessarily all referring to the same embodiment.In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.”

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the scope of theimplementations. It should be understood that they have been presentedby way of example only, not limitation, and various changes in form anddetails may be made. Any portion of the apparatus and/or methodsdescribed herein may be combined in any combination, except mutuallyexclusive combinations. The implementations described herein can includevarious combinations and/or sub-combinations of the functions,components and/or features of the different implementations described.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the scope of theimplementations. It should be understood that they have been presentedby way of example only, not limitation, and various changes in form anddetails may be made. Any portion of the apparatus and/or methodsdescribed herein may be combined in any combination, except mutuallyexclusive combinations. The implementations described herein can includevarious combinations and/or sub-combinations of the functions,components and/or features of the different implementations described.

What is claimed is:
 1. A method of operating a system configured togenerate a virtual environment, the method comprising: activating atracking device and tracking a position of a user electronic device inan ambient environment; detecting a transition condition, includingdetecting an external input and generating a transition command when thedetected external input is recognized as a request from an externalparty to initiate a transition of the user electronic device between thevirtual environment and the ambient environment, or comparing a distancebetween the user electronic device and at least one boundary of atracking area tracked by the tracking device within the ambientenvironment and generating a transition command when the distancebetween the user electronic device and the at least one boundary of thetracking area is less than or equal to a preset threshold distance; andperforming a transition process to transition between the virtualenvironment generated by the user electronic device and the ambientenvironment in response to the detected transition condition.
 2. Themethod of claim 1, wherein an area of the ambient environment is greaterthan a tracking area of the tracking device, such that a system boundarycorresponds to a boundary of the tracking area of the tracking devicewithin the ambient environment, detecting a transition conditionincluding: comparing a distance between the user electronic device andthe boundary of the tracking area of the tracking device within theambient environment; and generating the transition command to initiatethe transition process when the distance between the user electronicdevice and the boundary of the tracking area is less than or equal to apreset threshold distance.
 3. The method of claim 1, wherein an area ofthe ambient environment is less than a tracking area of the trackingdevice, such that a system boundary corresponds to a physical boundaryof the ambient environment, detecting a transition condition including:comparing a distance between the user electronic device and the physicalboundary of the ambient environment; and generating the transitioncommand to initiate the transition process when the distance between theuser electronic device and the physical boundary of the ambientenvironment is less than or equal to a preset threshold distance.
 4. Themethod of claim 1, detecting an external input including detecting aphysical gesture implemented by an external party, an audio commandgenerated by an external party, an approach of an external party, or aneye gaze of an external party, and comparing a distance between the userelectronic device and at least one boundary of a tracking area trackedby the tracking device including determining at least one of adirection, a speed, or an acceleration of movement of the userelectronic device based on a comparison of a current position to atleast one previous position of the electronic device detected by thetracking device, and performing a transition process including:replacing virtual image output displayed on a display of the userelectronic device in the virtual environment with transition imageoutput displayed on the display, the transition image output includingat least one of a monochromatic display, a pass through image of theambient environment captured by a camera of the user electronic device,or a fading out of the virtual image output of the virtual environmentand a fading in of the transition image output, or replacing virtualaudio output generated by the user electronic device in the virtualenvironment with transition audio output, the transition audio outputincluding at least one of a preset signal or tone or a pass through ofambient sound captured by a microphone of the user electronic device, ora fading out of the audio output of the virtual environment and a fadingin of transition audio output, or generating at least one of anexternally directed visual indicator, an externally directed audibleindicator, or an externally directed physical indicator indicating to anexternal party that the transition process is complete.
 5. The method ofclaim 1, comparing a distance between the user electronic device and atleast one boundary of a tracking area tracked by the tracking devicealso including: comparing a current position of the user electronicdevice detected by the tracking device to at least one previous positionof the user electronic device detected by the tracking device; anddetermining at least one of a direction of movement of the userelectronic device, a speed of movement of the user electronic device, oran acceleration of movement of the user electronic device based on thecomparison of the current position to the at least one previousposition.
 6. The method of claim 5, performing a transition process totransition between the virtual environment generated by the userelectronic device and the ambient environment including: selecting atransition profile based on at least one of the determined direction,speed or acceleration of movement of the user electronic device,environmental parameters of the virtual environment generated by theuser electronic device, or preset user preferences.
 7. The method ofclaim 1, wherein performing a transition process includes: initiatingthe transition process in response to the transition command andperforming the transition process such that the transition between thevirtual environment and the ambient environment is complete at or beforethe user electronic device reaches the at least one boundary of thetracking area.
 8. The method of claim 7, performing the transitionprocess also including: detecting that the user electronic device hascrossed the at least one boundary and is outside of the tracking area ofthe tracking device; detecting that the user electronic device has movedfrom the outside of the tracking area and across the at least oneboundary of the tracking area and has re-entered the tracking area; andtransitioning back into the virtual environment when the tracking devicedetects that the user electronic device has re-entered the trackingarea.
 9. The method of claim 1, detecting a transition conditionincluding: detecting physical obstacle in a movement path of the userelectronic device in the ambient environment, within a tracking area ofthe tracking device within the ambient environment; comparing a distancebetween the user electronic device and the detected obstacle; andgenerating a transition command to initiate the transition process whenthe distance between the user electronic device and the detectedobstacle is less than or equal to a preset threshold distance.
 10. Themethod of claim 9, detecting a physical obstacle in a movement path ofthe user electronic device in the ambient environment, within a trackingarea of the tracking device within the ambient environment including:intermittently scanning the tracking area and capturing an image of thetracking area; comparing a current image of the tracking area to atleast one previous image of the tracking area captured by the trackingdevice; and detecting the physical obstacle in the movement path of theuser electronic device in the ambient environment based on at least onechange detected when comparing the current image of the tracking area tothe at least one previous image of the tracking area.
 11. The method ofclaim 1, detecting an external input including at least one of:detecting a physical gesture implemented by an external party, detectingan audio command generated by an external party, detecting an approachof an external party, or detecting an eye gaze of an external party; andwherein performing the transition process includes generating at leastone of a visual indicator, an audible indicator, or a physical indicatordirected to the external party indicating completion of the transitionprocess.
 12. The method of claim 11, detecting an audio commandcorresponding to an externally generated transition command including atleast one of detecting a voice command including at least one previouslyset keyword, or detecting an audio tone, corresponding to the externallygenerated transition command.
 13. The method of claim 1, performing atransition process including transitioning from the virtual environmentto the ambient environment, including at least one of: fading out avirtual image output of the virtual environment displayed on a displayof the user electronic device and fading in a transition image outputdisplayed on the display of the user electronic device; or fading out ofaudio output of the virtual environment output by the user electronicdevice and fading in of transition audio output by the user electronicdevice.
 14. The method of claim 13, wherein fading out a virtual imageoutput and fading in a transition image output includes replacing avirtual video output of the virtual world displayed on the display witha monochromatic display.
 15. The method of claim 13, fading out avirtual image output and fading in a transition image output includingreplacing a virtual video output of the virtual world displayed on thedisplay with a pass through image of the ambient environment captured byan imaging device of the user electronic device.
 16. The method of claim13, fading out a virtual image output and fading in a transition imageoutput including: transitioning the display from a non-transparent stateto a transparent state while fading out the virtual image outputdisplayed on the display such that the ambient environment is visiblethrough the display upon completion of the fade out of the virtual imageoutput.
 17. An audio visual system, comprising: a user electronic devicegenerating a virtual environment, the user electronic device operatingin and being movable within an ambient environment; a tracking device incommunication with the user electronic device, the tracking devicetracking a position and movement of the user electronic device in theambient environment; and a processor configured to: translate thedetected movement of the user electronic device in the ambientenvironment into virtual movement in the virtual environment generatedby the user electronic device; detect a transition condition, includingdetect an external input recognized as a request from an external partyfor transition of the user electronic device between the virtualenvironment and the ambient environment, or compare a distance betweenthe user electronic device and at least one boundary of a tracking areatracked by the tracking device within the ambient environment andgenerate a transition command when the distance between the userelectronic device and the at least one boundary of the tracking area isless than or equal to a preset threshold distance; and perform atransition between the virtual environment generated by the userelectronic device and the ambient environment in response to thedetected transition condition.
 18. The system of claim 17, wherein theprocessor is further configured to: transition out of the virtualenvironment generated by the user electronic device based on transitionparameters included in a transition profile corresponding to thedetected transition condition, including at least one of: replacingvirtual image output displayed on a display of the user electronicdevice in the virtual environment with transition image output displayedon the display based on the transition parameters included in thetransition profile, the transition image output including at least oneof a monochromatic display or a pass through image of the ambientenvironment captured by a camera of the user electronic device,replacing virtual audio output generated by the user electronic devicein the virtual environment with transition audio output, the transitionaudio output including at least one of a preset signal or tone or a passthrough of ambient sound captured by a microphone of the user electronicdevice, or generating at least one of an externally directed visualindicator, an externally directed audible indicator, or an externallydirected physical indicator indicating to an external party that thetransition process is complete.
 19. The system of claim 17, the externalinput including at least one of a physical gesture implemented by anexternal party, an audio command generated by an external party, anapproach of an external party, or an eye gaze of an external party. 20.The system of claim 17, wherein the processor is further configured to:perform a transition from the virtual environment generated by the userelectronic device to the ambient environment such that the transitionout of the virtual environment is complete at or before the userelectronic device reaches the boundary; and perform a second transitionfrom the ambient environment into the virtual environment when thetracking device detects that the user electronic device has crossed theboundary and has re-entered the tracking area.
 21. A method, comprising:generating, by an electronic device, a virtual environment, theelectronic device operating in an ambient environment; tracking, by atracking device, a position of the electronic device in the ambientenvironment; detecting a transition condition, including: detecting anexternal input and recognizing the detected external input as a requestfrom an external party to initiate a transition, or detecting a distancebetween the electronic device and at least one boundary of a trackingarea tracked by the tracking device that is less than or equal to apreviously set threshold distance, including determining at least one ofa direction, a speed, or an acceleration of movement of the electronicdevice based on a comparison of a current position to at least oneprevious position of the electronic device detected by the trackingdevice; and performing a transition process to transition between thevirtual environment generated by the electronic device and the ambientenvironment in response to the detected transition condition, including:replacing virtual image output displayed on a display of the userelectronic device in the virtual environment with transition imageoutput displayed on the display, the transition image output includingat least one of a monochromatic display, a pass through image of theambient environment captured by a camera of the electronic device, or afading out of the virtual image output of the virtual environment and afading in of the transition image output, or replacing virtual audiooutput generated by the electronic device in the virtual environmentwith transition audio output, the transition audio output including atleast one of a preset signal or tone or a pass through of ambient soundcaptured by a microphone of the electronic device, or a fading out ofthe audio output of the virtual environment and a fading in oftransition audio output, or generating at least one of an externallydirected visual indicator, an externally directed audible indicator, oran externally directed physical indicator indicating to an externalparty that the transition process is complete.